{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using a Multiple Layer Network\n",
    "\n",
    "\n",
    "We will illustrate how to use a Multiple Layer Network in TensorFlow\n",
    "\n",
    "### Low Birthrate data:\n",
    "```\n",
    "#Columns    Variable                                      Abbreviation\n",
    "#---------------------------------------------------------------------\n",
    "# Low Birth Weight (0 = Birth Weight >= 2500g,            LOW\n",
    "#                          1 = Birth Weight < 2500g)\n",
    "# Age of the Mother in Years                              AGE\n",
    "# Weight in Pounds at the Last Menstrual Period           LWT\n",
    "# Race (1 = White, 2 = Black, 3 = Other)                  RACE\n",
    "# Smoking Status During Pregnancy (1 = Yes, 0 = No)       SMOKE\n",
    "# History of Premature Labor (0 = None  1 = One, etc.)    PTL\n",
    "# History of Hypertension (1 = Yes, 0 = No)               HT\n",
    "# Presence of Uterine Irritability (1 = Yes, 0 = No)      UI\n",
    "# Birth Weight in Grams                                   BWT\n",
    "#---------------------------------------------------------------------\n",
    "```\n",
    "The multiple neural network layer we will create will be composed of three fully connected hidden layers, with node sizes 50, 25, and 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import csv\n",
    "import os\n",
    "import os.path\n",
    "import random\n",
    "import numpy as np\n",
    "import random\n",
    "import requests\n",
    "from tensorflow.python.framework import ops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtain the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# name of data file\n",
    "birth_weight_file = 'birth_weight.csv'\n",
    "\n",
    "# download data and create data file if file does not exist in current directory\n",
    "if not os.path.exists(birth_weight_file):\n",
    "    birthdata_url = 'https://github.com/nfmcclure/tensorflow_cookbook/raw/master/01_Introduction/07_Working_with_Data_Sources/birthweight_data/birthweight.dat'\n",
    "    birth_file = requests.get(birthdata_url)\n",
    "    birth_data = birth_file.text.split('\\r\\n')\n",
    "    birth_header = birth_data[0].split('\\t')\n",
    "    birth_data = [[float(x) for x in y.split('\\t') if len(x)>=1] for y in birth_data[1:] if len(y)>=1]\n",
    "    with open(birth_weight_file, \"w\") as f:\n",
    "        writer = csv.writer(f)\n",
    "        writer.writerows([birth_header])\n",
    "        writer.writerows(birth_data)\n",
    "        f.close()\n",
    "\n",
    "# read birth weight data into memory\n",
    "birth_data = []\n",
    "with open(birth_weight_file, newline='') as csvfile:\n",
    "    csv_reader = csv.reader(csvfile)\n",
    "    birth_header = next(csv_reader)\n",
    "    for row in csv_reader:\n",
    "        birth_data.append(row)\n",
    "\n",
    "birth_data = [[float(x) for x in row] for row in birth_data]\n",
    "\n",
    "\n",
    "# Extract y-target (birth weight)\n",
    "y_vals = np.array([x[8] for x in birth_data])\n",
    "\n",
    "# Filter for features of interest\n",
    "cols_of_interest = ['AGE', 'LWT', 'RACE', 'SMOKE', 'PTL', 'HT', 'UI']\n",
    "x_vals = np.array([[x[ix] for ix, feature in enumerate(birth_header) if feature in cols_of_interest] for x in birth_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model\n",
    "\n",
    "Here we reset any graph in memory and then start to create our graph and vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# reset the graph for new run\n",
    "ops.reset_default_graph()\n",
    "\n",
    "# Create graph session \n",
    "sess = tf.Session()\n",
    "\n",
    "# set batch size for training\n",
    "batch_size = 150\n",
    "\n",
    "# make results reproducible\n",
    "seed = 3\n",
    "np.random.seed(seed)\n",
    "tf.set_random_seed(seed)\n",
    "\n",
    "# Split data into train/test = 80%/20%\n",
    "train_indices = np.random.choice(len(x_vals), round(len(x_vals)*0.8), replace=False)\n",
    "test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices)))\n",
    "x_vals_train = x_vals[train_indices]\n",
    "x_vals_test = x_vals[test_indices]\n",
    "y_vals_train = y_vals[train_indices]\n",
    "y_vals_test = y_vals[test_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we scale our dataset by the min/max of the _training set_.  We start by recording the mins and maxs of the training set. (We use this on scaling the test set, and evaluation set later on)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Record training column max and min for scaling of non-training data\n",
    "train_max = np.max(x_vals_train, axis=0)\n",
    "train_min = np.min(x_vals_train, axis=0)\n",
    "\n",
    "# Normalize by column (min-max norm to be between 0 and 1)\n",
    "def normalize_cols(mat, max_vals, min_vals):\n",
    "    return (mat - min_vals) / (max_vals - min_vals)\n",
    "\n",
    "x_vals_train = np.nan_to_num(normalize_cols(x_vals_train, train_max, train_min))\n",
    "x_vals_test = np.nan_to_num(normalize_cols(x_vals_test, train_max, train_min))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define our varibles, bias, and placeholders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define Variable Functions (weights and bias)\n",
    "def init_weight(shape, st_dev):\n",
    "    weight = tf.Variable(tf.random_normal(shape, stddev=st_dev))\n",
    "    return(weight)\n",
    "    \n",
    "\n",
    "def init_bias(shape, st_dev):\n",
    "    bias = tf.Variable(tf.random_normal(shape, stddev=st_dev))\n",
    "    return(bias)\n",
    "    \n",
    "    \n",
    "# Create Placeholders\n",
    "x_data = tf.placeholder(shape=[None, 7], dtype=tf.float32)\n",
    "y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define our model!  We start with a function that creates a fully connected later according to our variable specifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a fully connected layer:\n",
    "def fully_connected(input_layer, weights, biases):\n",
    "    layer = tf.add(tf.matmul(input_layer, weights), biases)\n",
    "    return(tf.nn.relu(layer))\n",
    "\n",
    "\n",
    "#--------Create the first layer (50 hidden nodes)--------\n",
    "weight_1 = init_weight(shape=[7, 25], st_dev=10.0)\n",
    "bias_1 = init_bias(shape=[25], st_dev=10.0)\n",
    "layer_1 = fully_connected(x_data, weight_1, bias_1)\n",
    "\n",
    "#--------Create second layer (25 hidden nodes)--------\n",
    "weight_2 = init_weight(shape=[25, 10], st_dev=10.0)\n",
    "bias_2 = init_bias(shape=[10], st_dev=10.0)\n",
    "layer_2 = fully_connected(layer_1, weight_2, bias_2)\n",
    "\n",
    "\n",
    "#--------Create third layer (5 hidden nodes)--------\n",
    "weight_3 = init_weight(shape=[10, 3], st_dev=10.0)\n",
    "bias_3 = init_bias(shape=[3], st_dev=10.0)\n",
    "layer_3 = fully_connected(layer_2, weight_3, bias_3)\n",
    "\n",
    "\n",
    "#--------Create output layer (1 output value)--------\n",
    "weight_4 = init_weight(shape=[3, 1], st_dev=10.0)\n",
    "bias_4 = init_bias(shape=[1], st_dev=10.0)\n",
    "final_output = fully_connected(layer_3, weight_4, bias_4)\n",
    "\n",
    "# Declare loss function (L1)\n",
    "loss = tf.reduce_mean(tf.abs(y_target - final_output))\n",
    "\n",
    "# Declare optimizer\n",
    "my_opt = tf.train.AdamOptimizer(0.025)\n",
    "train_step = my_opt.minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we initialize all the variables and start the training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation: 25. Loss = 15990.673\n",
      "Generation: 50. Loss = 4621.7324\n",
      "Generation: 75. Loss = 2795.2063\n",
      "Generation: 100. Loss = 2217.2898\n",
      "Generation: 125. Loss = 2295.652\n",
      "Generation: 150. Loss = 2047.0444\n",
      "Generation: 175. Loss = 1915.7662\n",
      "Generation: 200. Loss = 1732.4612\n",
      "Generation: 225. Loss = 1684.788\n",
      "Generation: 250. Loss = 1576.6499\n",
      "Generation: 275. Loss = 1579.0377\n",
      "Generation: 300. Loss = 1456.1995\n",
      "Generation: 325. Loss = 1521.6514\n",
      "Generation: 350. Loss = 1294.7653\n",
      "Generation: 375. Loss = 1507.5607\n",
      "Generation: 400. Loss = 1221.8282\n",
      "Generation: 425. Loss = 1636.6686\n",
      "Generation: 450. Loss = 1306.6858\n",
      "Generation: 475. Loss = 1564.3484\n",
      "Generation: 500. Loss = 1360.8761\n"
     ]
    }
   ],
   "source": [
    "# Initialize Variables\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "\n",
    "# Training loop\n",
    "loss_vec = []\n",
    "test_loss = []\n",
    "for i in range(500):\n",
    "    rand_index = np.random.choice(len(x_vals_train), size=batch_size)\n",
    "    rand_x = x_vals_train[rand_index]\n",
    "    rand_y = np.transpose([y_vals_train[rand_index]])\n",
    "    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})\n",
    "\n",
    "    temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})\n",
    "    loss_vec.append(temp_loss)\n",
    "    \n",
    "    test_temp_loss = sess.run(loss, feed_dict={x_data: x_vals_test, y_target: np.transpose([y_vals_test])})\n",
    "    test_loss.append(test_temp_loss)\n",
    "    if (i+1) % 25 == 0:\n",
    "        print('Generation: ' + str(i+1) + '. Loss = ' + str(temp_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is code that will plot the loss by generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEWCAYAAACufwpNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4FVX6wPHvmx5CIJGEGpCqNDGE0JQVRVTARXTXgopYQFwbKjbcnytY145dFwXFCqyVtSH2RguCSCcUIdQQQGoCSd7fH3MCV0iD5N6b8n6eZ57MnHNm5p1LuG/OlDOiqhhjjDHlISTYARhjjKk6LKkYY4wpN5ZUjDHGlBtLKsYYY8qNJRVjjDHlxpKKMcaYcmNJxVQZIpIoIktEJDqA+7xRRB4J1P6qIhH5i4gsDXYcpnxYUjHlSkRWi0jvIO1+JPCaqu51sXwrIioiJ/o2EpEPXPmpbjlORMaLyEYR2Skiy0RkpE97FZHdIrLLZ7rDVb8MXCoidQNziIcTkQgRuUdElro414nIZyJyZrBiKo77PFsWLKvqD6p6fDBjMuXHkoqpEkQkErgcePOQqmXAYJ92dYDuQKZPmzFATaANUBs4B0g/ZDsnqmpNn+lRAFXNBj7z3Ye/iEhYEVXvAgNcDPFAM+Bp4Gx/x3SoYmI01YQlFRMwInK1iKSLyFYRmSIiDV25iMgYEdksIjtE5DcRae/q+onIIteDWCcitxWx+a7AdlXNOKT8LeAiEQl1yxcDHwD7fNp0Bt5W1W2qmq+qS1T13SM4tG8p5gvc/WU+XERWisgWEXlMREJ86q8SkcUisk1EporIsYese72ILAeWF7Lt3sAZwABVnamq+9z0uare5NOuoYi8JyKZIrJKRIb71I0Wkcki8rr7nBeKSOoRrPuuiLwpIjuAK0Ski4hMF5HtIrJBRJ4TkQjX/nu36q+ux3eRiJwqIhk+22zjepnbXSzn+NS9JiLPi8gnLtaZItKixH8hEzCWVExAiEgv4N/AhUAD4Hdgoqs+EzgFOA6vp3AhkOXqxgHXqGos0B74uohdnAAUdl5+PbDI7QO8v+ZfP6TNDOBBEblSRFod2ZEBsBg4sYQ25wGpQAper+IqABEZAPwT+BuQCPwAvHPIuufiJc22hWy3NzCzkGR6gEtg/wN+BRoBpwM3i8hZPs3Owfv3iAOmAM8dwboD8HpLcXhJPA+4BUjA6xWeDlwHoKqnuHUKen6TDok13O3vC6AucCPwloj4nh4bCNyL1ytLBx4s6thN4FlSMYFyKTBeVX9R1RzgLqC7iDQF9gOxQGtAVHWxqm5w6+0H2opILdeT+KWI7ccBO4uoex0YLCKtgThVnX5I/Y14X4Y3AItcb6rvIW1+cX85F0y+X6o78ZJhcR5R1a2qugZ4Cq/HBPAP4N/umHOBh4Bk396Kq99acK3oEAnAxoIFETnGxfeHiGS74s5Aoqre53oxK/GuBQ302c6PqvqpquYBb3AwSZZm3emq+qHr5e1V1TmqOkNVc1V1NfAfoGcJn0+BbninIh92+/sa+Njn8wL4QFVnuc/rLSC5lNs2AWBJxQRKQ7zeCQCquguvN9LIfXE8BzwPbBaRsSJSyzX9O9AP+F1EvhOR7kVsfxteYirM+0AvvKTxxqGV7ovwIVXtBNQBJgP/FZFjfJqlqGqczzTVpy4W+KPYo4e1PvO/430eAMcCTxckK2ArIHi9gsLWPVQWXs+v4Fi2qmoc0AmI9NlHQ9+kiNc7queznY0+83uAKHd9pDTr/ik+ETlORD4W78aHHXiJMqGYY/DVEFirqvk+Zb/z58/j0FhrlnLbJgAsqZhAWY/3BQWAiMTgfYGvA1DVZ9yXelu802C3u/LZqjoA71TIh3hf+IWZ79Y7jKruwbuYfi2FJJVD2hZ8CcbgXfAujTZ4p4eK09hnvgne5wHeF/I1hySsaFX92TesYrb7FdBZRJKKabMWWHXIPmJVtV8JMZd23UPjexFYArRS1Vp4SUhKsS/wPpfGvtec8D6vdaVc3wSZJRXjD+EiEuUzheFdJ7hSRJLFu1PrIbxrAatFpLOIdHXn03cD2UC+eLfKXioitVV1P7ADyC9in7OAOBFpVET9P4Ge7nTMn4jIv1wMESISBdwEbKfwazSF6YmXtIpzu4jEi0hjt/2CawkvAXeJSDsXS20RuaCU+0VVvwC+AT50n2GE+xy7+TSbBewUkTtFJFpEQkWkvYh0LsUujmbdWLx/q13ulOO1h9RvApoXse5MvN7HHSISLt5t3/05eP3NVHCWVIw/fArs9ZlGq+qXwL+A94ANQAsOnpevhXeefhveqY4s4DFXdxmw2p1G+QfetZnDqOo+4DVgUBH161X1xyLiVeBVYAveX8pnAGe7U3QFCu5WKpieAnBJqB8wochPw/MRMAeYB3yCdwMCqvoB8Agw0R3jAuDQ6zklOQ/vusObeMlwFd7ndJbbRx7wV7xrD6vccb5CydeBjnbd24BL8K41vczBBFpgNDDBnU678JD97cNLIn3dvl4ABqvqkpJiNRWD2Eu6TFUhIgV3T3Us4qK2P/Z5I9BYVe8opo3inQo69NkXY6ocSyrG+JklFVOd2OkvY4wx5cZ6KsYYY8qN9VSMMcaUm2o3+FtCQoI2bdo02GEYY0ylMmfOnC2qmlhSu2qXVJo2bUpaWlqwwzDGmEpFRH4vuZWd/jLGGFOOLKkYY4wpN5ZUjDHGlJtqd03FGFM17N+/n4yMDLKzs0tubEotKiqKpKQkwsPDj2p9SyrGmEopIyOD2NhYmjZtikhpB0E2xVFVsrKyyMjIoFmz0g7S/Wd2+ssYUyllZ2dTp04dSyjlSESoU6dOmXp/llSMMZWWJZTyV9bP1JJKKT377LNMmnToCN7GGGN8WVIppbFjx1pSMcYckJWVRXJyMsnJydSvX59GjRodWN63b1+ptnHllVeydGlp3wUHr7zyCjfffPPRhhwQdqG+lGrUqMGePXuCHYYxpoKoU6cO8+bNA2D06NHUrFmT22677U9tVBVVJSSk8L/fX331Vb/HGWjWUymlmJgYdu/eHewwjDEVXHp6Om3btuXSSy+lXbt2bNiwgWHDhpGamkq7du247777DrTt0aMH8+bNIzc3l7i4OEaOHMmJJ55I9+7d2bx5c6n3+eabb3LCCSfQvn17/vnPfwKQm5vLZZdddqD8mWeeAWDMmDG0bduWDh06MGhQoS9KLRPrqZRSjRo1WL9+fbDDMMYU4uabbz7QaygvycnJPPXUU0e17pIlS3j99ddJTU0F4OGHH+aYY44hNzeX0047jfPPP5+2bdv+aZ0//viDnj178vDDDzNixAjGjx/PyJEjS9xXRkYGd999N2lpadSuXZvevXvz8ccfk5iYyJYtW/jtt98A2L59OwCPPvoov//+OxEREQfKypPfeyoiEioic0XkY7fcTERmiki6iEwSkQhXHumW0119U59t3OXKl4rIWT7lfVxZuoiU/OmXQUxMjJ3+MsaUSosWLQ4kFIB33nmHlJQUUlJSWLx4MYsWLTpsnejoaPr27QtAp06dWL16dan2NXPmTHr16kVCQgLh4eFccsklfP/997Rs2ZKlS5cyfPhwpk6dSu3atQFo164dgwYN4q233jrqBxyLE4ieyk3AYqCWW34EGKOqE0XkJWAI8KL7uU1VW4rIQNfuIhFpCwwE2gENgS9F5Di3reeBM4AMYLaITFHVw/+1ykGX7duJyMryx6aNMWV0tD0Kf4mJiTkwv3z5cp5++mlmzZpFXFwcgwYNKvQ5kIiIiAPzoaGh5ObmlimGOnXqMH/+fD777DOef/553nvvPcaOHcvUqVP57rvvmDJlCg899BDz588nNDS0TPvy5deeiogkAWcDr7hlAXoB77omE4Bz3fwAt4yrP921HwBMVNUcVV0FpANd3JSuqitVdR8w0bX1i3MWLOBaP3QVjTFV244dO4iNjaVWrVps2LCBqVOnluv2u3btyjfffENWVha5ublMnDiRnj17kpmZiapywQUXcN999/HLL7+Ql5dHRkYGvXr14tFHH2XLli3lfgbG3z2Vp4A7gFi3XAfYrqoFKTgDaOTmGwFrAVQ1V0T+cO0bATN8tum7ztpDyrsWFoSIDAOGATRp0uSoDmR3rVrU3bDhqNY1xlRfKSkptG3bltatW3Psscdy8sknl2l748aN49133z2wnJaWxv3338+pp56KqtK/f3/OPvtsfvnlF4YMGYKqIiI88sgj5Obmcskll7Bz507y8/O57bbbiI2NLWZvR85v76gXkb8C/VT1OhE5FbgNuAKYoaotXZvGwGeq2l5EFgB9VDXD1a3ASxKj3TpvuvJxwGduN31Udagrvwzoqqo3FBdXamqqHs1LuqafdBIdpk8nav9+QsPs/gZjgm3x4sW0adMm2GFUSYV9tiIyR1VTi1jlAH9+O54MnCMi/YAovGsqTwNxIhLmeitJwDrXfh3QGMgQkTCgNpDlU17Ad52iystdzjHHEAPs3LSJ2EaNSmxvjDHVkd+uqajqXaqapKpN8S60f62qlwLfAOe7ZpcDH7n5KW4ZV/+1et2oKcBAd3dYM6AVMAuYDbRyd5NFuH1M8dfx5NatC0D2ypX+2oUxxlR6wXj48U5ghIik410zGefKxwF1XPkIYCSAqi4EJgOLgM+B61U1z/V0bgCm4t1dNtm19YvNqakcD+xITPTXLowxptILyMUBVf0W+NbNr8S7c+vQNtnABUWs/yDwYCHlnwKflmOoRYqoW5dlwJ79+wOxO2OMqZRsmJZSqhEZyU1AyI8/BjsUY4ypsCyplFJMbCwPAjW//jrYoRhjTIVlSaWUatWuzQYgZNOmYIdijKkAymPoe4Dx48ezcePGQusGDRrEhx9+WF4hB4Q9cFFKcXFxrAVaHcHIocaYqqs0Q9+Xxvjx40lJSaF+/frlHWJQWE+llOLi4tgAhG3ezOjRo/HXQ6PGmMpvwoQJdOnSheTkZK677jry8/MLHYp+0qRJzJs3j4suuqjUPZz8/HxGjBhB+/btOeGEEw48Xb9u3Tp69OhBcnIy7du35+effy5y+Ht/sp5KKdV2p7+it23j3nvv5dJLL6VVq1bBDssYU+DUUw8vu/BCuO462LMH+vU7vP6KK7xpyxY4//w/13377VGFsWDBAj744AN+/vlnwsLCGDZsGBMnTqRFixaHDUUfFxfHs88+y3PPPUdycnKptv/f//6XxYsX8+uvv5KZmUnnzp055ZRTePPNN+nfvz933nkneXl57N27lzlz5hQ6/L0/WU+llEJCQhhTqxYFHdQwG6rFGFOIL7/8ktmzZ5OamkpycjLfffcdK1asKHIo+iP1448/cvHFFxMaGkr9+vXp0aMHaWlpdO7cmVdeeYV7772XBQsWULNmzXLb55Gwb8YjIPHx7N6xA4D99ryKMRVLcT2LGjWKr09IOOqeyaFUlauuuor777//sLrChqIvL7169eLbb7/lk08+YfDgwdxxxx1ceumlft1nYayncgSOr1GDx4A2UOZ3HRhjqqbevXszefJktmzZAnh3ia1Zs6bQoegBYmNj2blzZ6m3/5e//IWJEyeSn5/Ppk2b+Omnn0hNTeX333+nfv36DBs2jCuvvJK5c+cWuU9/sp7KEcjdvp3bgDlYUjHGFO6EE05g1KhR9O7dm/z8fMLDw3nppZcIDQ09bCh6gCuvvJKhQ4cSHR3NrFmz/vSyLoChQ4dyww3e4OvNmjXju+++Y8aMGXTo0AER4cknn6Ru3bqMHz+eJ598kvDwcGJjY3njjTdYu3Ztofv0J78NfV9RHe3Q9wBxImzHG5hs0Jw5pKSklGtsxpjSs6Hv/acsQ9/b6a8j0DIlhd14bwiznooxxhzOksoRmPbll+yvV48mWFIxxpjCWFI5AvHx8eQ3akQcdveXMRVBdTt9Hwhl/UwtqRyhhY8/zplYT8WYYIuKiiIrK8sSSzlSVbKysoiKijrqbdjdX0cozH3YllSMCa6kpCQyMjLIzMwMdihVSlRUFElJSUe9vt+SiohEAd8DkW4/76rqKBF5DegJ/OGaXqGq80RE8N5h3w/Y48p/cdu6HLjbtX9AVSe48k7Aa0A03su6blI//9lSe+lSJgKhNlqxMUEVHh5Os2bNgh2GOYQ/T3/lAL1U9UQgGegjIt1c3e2qmuymea6sL97751sBw4AXAUTkGGAU0BXvjZGjRCTerfMicLXPen38eDwAROzezUVAZEaGv3dljDGVjt+Sinp2ucVwNxXXixgAvO7WmwHEiUgD4CxgmqpuVdVtwDS8BNUAqKWqM1zv5HXgXH8dT4H8Ro0AiLQh8I0x5jB+vVAvIqEiMg/YjJcYZrqqB0VkvoiMEZFIV9YIWOuzeoYrK648o5DywuIYJiJpIpJW1vOv6s41RlhSMcaYw/g1qahqnqomA0lAFxFpD9wFtAY6A8cAd/ozBhfHWFVNVdXUxMTEMm0rpHZttgPRdnHQGGMOE5BbilV1O/AN0EdVN7hTXDnAq3jXSQDWAY19VktyZcWVJxVS7lfh4eHMB/L8vSNjjKmE/JZURCRRROLcfDRwBrDEXQvB3e11LrDArTIFGCyebsAfqroBmAqcKSLx7gL9mcBUV7dDRLq5bQ0GPvLX8RQICwujJ/DzRRf5e1fGGFPp+PM5lQbABBEJxUtek1X1YxH5WkQSAQHmAf9w7T/Fu504He+W4isBVHWriNwPzHbt7lPVrW7+Og7eUvyZm/yq4OVc9pyKMcYczkYpPkJbtmzh6sREXmjWjAYLFngv/zHGmCrORin2k7CwMGoCDVatQteuLbG9McZUJ5ZUjlBYWNiB+5tHXXVVUGMxxpiKxpLKEQoPDz+QVNb8/HNQYzHGmIrGksoRCgsLO/DEZWNs6G1jjPFlSeUIhYSEsA/voZvtwN69e4MckTHGVByWVI6Q90gM9AKeA/bs2RPUeIwxpiKxpFJGu3fvDnYIxhhTYVhSOUo3AwuB3bt2ldTUGGOqDUsqR0mAtkCOvazLGGMOsKRylH53P/NXrgxqHMYYU5FYUjlKqwtmfv+9mFbGGFO9WFI5Sqvdz1B7rbAxxhxgSeUobQXeB7bXrBnsUIwxpsKwpFIGfwcWt20b7DCMMabCsKRSRrt37gx2CMYYU2H4882PUSIyS0R+FZGFInKvK28mIjNFJF1EJolIhCuPdMvprr6pz7bucuVLReQsn/I+rixdREb661iK8gBw7X33BXq3xhhTYfmzp5ID9FLVE4FkoI97TfAjwBhVbQlsA4a49kOAba58jGuHiLQFBgLtgD7ACyIS6t4o+TzQF++RkYtd24DZHRpKzO7dsGNHIHdrjDEVlt+SinoKHjcPd5PiDZv1riufgPeeeoABbhlXf7p79/wAYKKq5qjqKrzXDXdxU7qqrlTVfcBE1zZgNkZFeTN2W7ExxgB+vqbiehTzgM3ANGAFsF1VC17wngE0cvONwHtViav/A6jjW37IOkWVFxbHMBFJE5G0zMzMMh/XunXr2LRpE5kFrxJevbrM2zTGmKrAr0lFVfNUNRlIwutZtPbn/oqJY6yqpqpqamJiYpm317BhQ+rWrcvWWrW8AuupGGMMEKC7v1R1O94rSLoDcSIS5qqSgHVufh3ee69w9bWBLN/yQ9Ypqjxg9sbG8smxx4LdVmyMMYB/7/5KFJE4Nx8NnAEsxksu57tmlwMfufkpbhlX/7V6r1WcAgx0d4c1A1oBs4DZQCt3N1kE3sX8Kf46nsLE1KzJE82bQ69egdytMcZUWGElNzlqDYAJ7i6tEGCyqn4sIouAiSLyADAXGOfajwPeEJF0vAfWBwKo6kIRmQwsAnKB61U1D0BEbgCmAqHAeFVd6MfjOUxMTAw7tm2DjRuhfv1A7toYYyokvyUVVZ0PdCykfCXe9ZVDy7OBC4rY1oPAg4WUfwp8WuZgj1KNGjX469Sp7D/uOMLttmJjjLEn6ssiJiaGVUD4zp2wbVuwwzHGmKCzpFIGNWrUYEXBwooVxTU1xphqwZJKGcTExFhSMcYYH5ZUyiAiIoKC9z7mLVsW1FiMMaYisKRSBtnZ2ewBbgW2d+oU7HCMMSboLKmUwe7duwF4EtjQpElwgzHGmArAkkoZFCSVeOCqE07Ae1bTGGOqL0sqZdChQwcAbgRmANs3bQpqPMYYE2yWVMrg9ttv54svvmAF3ge5dc6cYIdkjDFBZUmlDEJDQznjjDMY/tRTAOyePz/IERljTHBZUikHce7Or/1LlgQ5EmOMCS5LKuUgsW1bdgI/vv46O2wMMGNMNWZJpRzExcczCBgLpKWlBTscY4wJGksq5UBEmNekCYs4eJuxMcZUR5ZUysk3r73GQOAPG63YGFONWVIpJwlpabwD5K1ZE+xQjDEmaPz5OuHGIvKNiCwSkYUicpMrHy0i60Rknpv6+axzl4iki8hSETnLp7yPK0sXkZE+5c1EZKYrn+ReKxwUUe3aARC2enWwQjDGmKDzZ08lF7hVVdsC3YDrRaStqxujqslu+hTA1Q0E2gF9gBdEJNS9jvh5oC/QFrjYZzuPuG21BLYBQ/x4PMWKaOuFFLl2bbBCMMaYoPNbUlHVDar6i5vfCSwGGhWzygBgoqrmqOoqIB3vtcNdgHRVXamq+4CJwAAREaAX8K5bfwJwrn+OphQaNyYHL6nMnDkzaGEYY0wwBeSaiog0xXtffcG37Q0iMl9ExotIvCtrBPj+mZ/hyooqrwNsV9XcQ8oL2/8wEUkTkbTMzMxyOKJChIaSERHB/sWL6datG3v27PHPfowxpgLze1IRkZrAe8DNqroDeBFoASQDG4An/B2Dqo5V1VRVTU1MTPTbfu5p1Yob3Xx2drbf9mOMMRWVX5OKiITjJZS3VPV9AFXdpKp5qpoPvIx3egtgHdDYZ/UkV1ZUeRYQJyJhh5QHTWbDhqx385ZUjDHVkT/v/hJgHLBYVZ/0KW/g0+w8YIGbnwIMFJFIEWkGtAJmAbOBVu5Orwi8i/lT1Ht5yTfA+W79y4GP/HU8pdE2OprbgGOAvXv3BjMUY4wJirCSmxy1k4HLgN9EZJ4r+yfe3VvJgAKrgWsAVHWhiEwGFuHdOXa9quYBiMgNwFQgFBivqgvd9u4EJorIA8BcvCQWNB1jY7kcLxNaT8UYUx35Lamo6o+AFFL1aTHrPAg8WEj5p4Wtp6orOXj6LOhqpaTAW2/REksqxpjqyZ6oL0f1UlPZB5ZUjDHVliWVctSqTRtW4V0MsmsqxpjqyJJKOUpMTOSYLl1ogfVUjDHVkyWVcrbl8cfpiiUVY0z1VKqkIiItRCTSzZ8qIsNFJM6/oVVOEQ0bsh9LKsaY6qm0PZX3gDwRaYn3gsPGwNt+i6oSi8nK4mkgavnyYIdijDEBV9qkku/G2DoPeFZVbwcalLBOtRQZEsJwIC49PdihGGNMwJU2qewXkYvxnlr/2JWF+yekyi2iVSv2AzEbNwY7FGOMCbjSJpUrge7Ag6q6yg2j8ob/wqq8omrWZBVQa9OmYIdijDEBV6on6lV1ETAcwA1VH6uqj/gzsMoqNDSUFSJ02LIl2KEYY0zAlfbur29FpJaIHAP8ArwsIk+WtF51tTosjLD9+4MdhjHGBFxpT3/Vdu9C+Rvwuqp2BXr7L6zKbXTt2tx70UXBDsMYYwKutEklzA1ZfyEHL9SbIkTVqGHDtBhjqqXSJpX78IaeX6Gqs0WkOWAPYhShYVQU13zxBfzvf8EOxRhjAqpUSUVV/6uqHVT1Wre8UlX/7t/QKq/YBg1IXb8eZs4MdijGGBNQpb1QnyQiH4jIZje9JyJJJazTWES+EZFFIrJQRG5y5ceIyDQRWe5+xrtyEZFnRCRdROaLSIrPti537ZeLyOU+5Z1E5De3zjPubZNBVzcpiXVhYWAPQBpjqpnSnv56Fe91vw3d9D9XVpxc4FZVbQt0A64XkbbASOArVW0FfOWWAfrijRrfChgGvAheEgJGAV3xXsg1qiARuTZX+6zXp5TH41cNGjRgaX4+mp7O1q1b2blzZ7BDMsaYgChtUklU1VdVNddNrwGJxa2gqhtU9Rc3vxNYDDQCBgATXLMJwLlufgDenWWqqjOAOHdzwFnANFXdqqrbgGlAH1dXS1VnuPfVv+6zraBq0KABy/LzYfly6tSpQ/PmzYMdkjHGBERpk0qWiAwSkVA3DQKySrsTEWkKdARmAvVUdYOr2gjUc/ONgLU+q2W4suLKMwopL2z/w0QkTUTSMjMzSxv2UWvQoAHzgOymTakBbLEHIY0x1URpk8pVeLcTbwQ2AOcDV5RmRRGpiTfK8c3uWZcDXA9DSxvs0VLVsaqaqqqpiYnFdrDKRaNGjRgH1Jg/nz1+35sxxlQcpb3763dVPUdVE1W1rqqeC5R495eIhOMllLdU9X1XvMmdusL93OzK1+ENqV8gyZUVV55USHnQtWvXLtghGGNMUJTlzY8jiqt0d2KNAxarqu+QLlPwRjvG/fzIp3ywuwusG/CHO002FThTROLdBfozgamuboeIdHP7GuyzraCqU6cOAkwHbg92MMYYE0ClGlCyCCXdvnsycBnwm4jMc2X/BB4GJovIEOB3vNNqAJ8C/YB0YA/eyMio6lYRuR+Y7drdp6pb3fx1wGtANPCZmyoEBeoCycEOxBhjAki8yxpHsaLIGlVtUs7x+F1qaqqmpaX5fT/z589n44knEod3L/TRfs7GGFMRiMgcVU0tqV2xPRUR2UnhF9IFr3dgitChQwcaXnABIf/9LwkJCcEOxxhjAqLYayqqGquqtQqZYlW1LKfOqoWE7t05Bqhlw+AbY6qJslyoNyXp1In5xx1HSE5OsCMxxpiAsKTiT6ecwrsXXcTKffvsmooxplqwpOJnkZGRkJ9Pbm5usEMxxhi/s+sifnbVf/7DsUBOTg7h4eHBDscYY/zKeip+tj8mhtZAdnZ2sEMxxhi/s6TiZzsaNfKSyh4bBcwYU/VZUvGzXU2aUBPYPGdOsEMxxhi/s6TiZ3ubNgXg//72N7sDzBhT5VlS8bPsli15Cm/45FWrVgU7HGOM8StLKn4WmpDALcBvQCDGHDPGmGCypOJn4eHhRADNgHnz5pX86MOyAAAexElEQVTU3BhjKjVLKn62ZcsWxgC/AOsyMkpqbowxlZolFT877bTT2BgfTxyw166pGGOqOEsqfpaQkMB9774LQNTKlYwYMYINGzYEOSpjjPEPvyUVERkvIptFZIFP2WgRWSci89zUz6fuLhFJF5GlInKWT3kfV5YuIiN9ypuJyExXPklEIvx1LGXWti0AtdevZ8yYMQwfPjzIARljjH/4s6fyGtCnkPIxqprspk8BRKQtMBBo59Z5QURCRSQUeB7oC7QFLnZtAR5x22oJbAOG+PFYyqZePfZGR9PeLe6396sYY6oovyUVVf0e2FpiQ88AYKKq5qjqKrz31HdxU7qqrlTVfcBEYICICNALeNetPwE4t1wPoDyJkHb55bzqFm1gSWNMVRWMayo3iMh8d3os3pU1Atb6tMlwZUWV1wG2q2ruIeWFEpFhIpImImmZmZnldRxHJH/gQGa6eUsqxpiqKtBJ5UWgBZAMbACeCMROVXWsqqaqampiYmIgdnmYZvXr0xeoF5S9G2NMYAQ0qajqJlXNU9V84GW801vgjWLS2KdpkisrqjwLiBORsEPKK6yGeXl8ineeb/v27cEOxxhj/CKgSUVEGvgsngcU3Bk2BRgoIpEi0gxoBcwCZgOt3J1eEXgX86eoNzLjN8D5bv3LgY8CcQxHK6xNG9YAZ2BJxRhTdfnzluJ3gOnA8SKSISJDgEdF5DcRmQ+cBtwCoKoLgcnAIuBz4HrXo8kFbgCmAouBya4twJ3ACBFJx7vGMs5fx1IuRJgXG8tJwLatpb1/wRhjKhepbsOxp6amarAGdtRnnkFuuolOiYnM2bw5KDEYY8zREJE5qppaUjt7oj6ApFs3AI7bti3IkRhjjH9YUgmk5GReGT6cybm59s56Y0yVZEklkCIiyG3Thnxg9OjR5OfnBzsiY4wpV5ZUAqz5li08CzzxyCNMmTIl2OEYY0y5sqQSYPV27eIGoD02BpgxpuqxpBJg2sV73rMr4A1hZowxVYcllQCr0a4dmXhJZceOHcEOxxhjypUllQCLi49nJl5SsSfrjTFVjSWVAIuLi2MmEAXsysoKdjjGGFOuLKkEWEREBA/hDdX87fTp7Nu3L9ghGWNMubGkEgRjnn4agG+++YZ77rknyNEYY0z5saQSBMOHD+dxvJfLzJgxI9jhGGNMubGkEiQJeO8/rhkTE+xQjDGm3FhSCZINjRtTH8hbvTrYoRhjTLmxpBIkt02eDEBdSyrGmCrEny/pGi8im0VkgU/ZMSIyTUSWu5/xrlxE5BkRSReR+SKS4rPO5a79chG53Ke8k3vhV7pbt1I9nh6WkkJuWBgn7NnDNhsK3xhTRfizp/Ia0OeQspHAV6raCvjKLQP0xXuFcCtgGN41bETkGGAU3rOCXYBRBYnItbnaZ71D91WxRUSQ0bMna4GVK1cGOxpjjCkXfksqqvo9cOh7cwcAE9z8BLxr1QXlr6tnBhDn3md/FjBNVbeq6jZgGtDH1dVS1RnuffWv+2yr0tj++OM8A3Tu3Jmt9ophY0wVEOhrKvVUdYOb3wjUc/ONgLU+7TJcWXHlGYWUVyrNmjUjFIhQZdq0acEOxxhjyixoF+pdD0MDsS8RGSYiaSKSlpmZGYhdlkrtPXv4A7gcyMvLC3Y4xhhTZoFOKpvcqSvcz82ufB3Q2KddkisrrjypkPJCqepYVU1V1dTExMQyH0S5qV+fkJo16QZk2ThgxpgqINBJZQreH+a4nx/5lA92d4F1A/5wp8mmAmeKSLy7QH8mMNXV7RCRbu6ur8E+26o8RIg87TR6YEnFGFM1+POW4neA6cDxIpIhIkOAh4EzRGQ50NstA3wKrATSgZeB6wBUdStwPzDbTfe5MlybV9w6K4DP/HUs/hRyyim0AvavXVtiW2OMqejEu7RRfaSmpmpaWlqwwzhoxgzo3p0htWrx+OrVxMfHl7yOMcYEmIjMUdXUktrZE/XBlpLCv6Oi+GnHDoYPHx7saIwxpkzCgh1AtRcRwT+zswFI2rChhMbGGFOxWU+lArh35EjOBGrs28fGjRuDHY4xxhw1SyoVwD0DBjAViP7hBxo0aGAPQhpjKi1LKhVBaip7IyM5wy3+9ttvQQ3HGGOOliWViiAsjN3dutEvPByAzZs3l7CCMcZUTJZUKoiEgQNpuH8/vZs0YbW9Y8UYU0lZUqkozjwTgJ55eUyaNIlPP/00yAEZY8yRs6RSUTRvDr/8wpbzzgPggw8+CHJAxhhz5CypVCQdO/LwY48RGxvLnj17gh2NMcYcMUsqFUl+PlF33sn98fFUpCH6jTGmtCypVCQhIbBkCZds3sz6tWvJzc0NdkTGGHNELKlUNEOHkpidTdKSJXTv3j3Y0RhjzBGxpFLRnHMOu6KiuBqoUKMpG2NMKVhSqWgiI/kqKYlzgAZAfn5+sCMyxphSs6RSAbUaM4Y3AAE+//zzYIdjjDGlFpSkIiKrReQ3EZknImmu7BgRmSYiy93PeFcuIvKMiKSLyHwRSfHZzuWu/XIRubyo/VU2bf/6Vxp9/jnrgbPPPpvq9iI1Y0zlFcyeymmqmuzzJrGRwFeq2gr4yi0D9AVauWkY8CJ4SQgYBXQFugCjChJRVZCcnMyJwDXAE088wUsvvRTskIwxpkQV6fTXAGCCm58AnOtT/rp6ZgBxItIAOAuYpqpbVXUbMA3oE+ig/aVevXpM6duX54Dxt9/Otddey969e4MdljHGFCtYSUWBL0RkjogMc2X1VLXg1YcbgXpuvhGw1mfdDFdWVPlhRGSYiKSJSFpleqhw37/+xS5gjFv++OOPycvLC2ZIxhhTrGAllR6qmoJ3aut6ETnFt1K9iwjldiFBVceqaqqqpiYmJpbXZv0uqWNHRuN1yc4GLrzwQi688MLgBmWMMcUISlJR1XXu52bgA7xrIpvcaS3cz4KXiqwDGvusnuTKiiqvMqKiongpJITFwDNATeD9998n273T3hhjKpqAJxURiRGR2IJ54ExgATAFKLiD63LgIzc/BRjs7gLrBvzhTpNNBc4UkXh3gf5MV1aldOzShauBt4EcV/bQQw8FMSJjjClaMHoq9YAfReRXYBbwiap+DjwMnCEiy4HebhngU2AlkA68DFwHoKpbgfuB2W66z5VVKW+//TZNLr6Y7884g/3AWaefzssvv0yXLl0YOHAga9euLXEbxhgTKFLdnoFITU3Vyjj8ya5du8icOpW4a6/llMxMFvjUVbd/Q2NM4InIHJ9HQIpUkW4pNsWoWbMmzTp3pkZ+Pl8AbXzq7N0rxpiKwpJKZdKkCdn/+x8AP4WF8c+TTwZg2bJlLF++nE8++SSY0RljDGHBDsAcmdrdu6OzZlH7sst4YMYMpgMdO3Y8UJ+Tk0NERETwAjTGVGvWU6mE4jp3RmbNYv8tt/CjK+sANAEiIyMZN24cCxYsYN++fUGM0hhTHdmF+kpu5syZ1IqMpOUZZxC+ZQvbgF+BecCXQKNhw7jxxhtp/8EHkJ8PBf/eqtCtG/TtC/v2wf33e+UhISDi/ezRA3r1gpwcmDQJatSA8HDIzfV+nnACNGt2cJsigf8AjDEBUdoL9ZZUqgJVdPZsnr3iCs6sW5f6mzYRtXw5b+blcbVrksfh3VK96Sbkqadg926oVQtVRXx/H0aNgtGjYcMGaNjw8P0+9hjcdhukp0PbtpCQAImJ3lS3Lgwfjnbtiixf7m0nNxfy8iA6GmJjYdgw6NgRMjJg6lSvrGZNqFXLm2/RwlvOz/cSliUtY4KmtEnFrqlUBSJIly4MX7ToQNHvq1bx7T//yeJRo3j//fdJevppNm7e/Kexb7pMn06dfv1YsmQJPQcPZs6cOfz2228A3HHbbVx/xRU8cv31fP/11/Tu35+m9eox/bvvSOnalUv+/neSunThgw8+YFd6On2vvJLstWuJ37+fGrt3k/Pjj3DmmZxzzz2kiHDvkiVs3LKF+g0aEJmfDzt3kt27N1EdO8Kvv8LQoYcf17Rp0Ls3vPceXHKJl2BiYyE2Fo2NRcaP95LZjz/CxIkH6oiO9qaBAyEuDlauhOXLvbKoqIM/mzWDsDDYtQv27j1YF2b/LYw5WtZTqSZ27drFhAkTmDlzJr///jtNmzZlzpw5rF+/nuOPP56ZM2cSEhJS6ICVNWrUICQkhF27dv2pPCkpiYyMjMPaN2nShDVr1hxYFhHatGnDokWLaNasGZMnT2bRokVcc8013HrrrfQ66SRidu+mTkQErzz1FGsXLiQxKooTb7iBs6+4gn1paST99BPs3Mn+rVtZNX8+GYsXEzNhAl0vuQRefRVuvRXduRPJzT0YSHq619t55BEYOfKwONm4EerVg3/9Cx544GB5aKiXXDZuhJgYeOgheOstiIz0pqgoL8FNmeL1niZNgrlzDyarqCivt3Xlld725syBrCyvPDzcS1rR0dC+vVe/bp13ijEy0uvl2Y0WpgKy019FqK5JpSR79uwhKiqKTZs2Ubt2bZYtW8bdd99NSkoK9957Lzk5OWzZsoVt27YxefJkZsyYwZo1a1i2bBkAl112GZ06dWLEiBEkJSXRq1cvXnvtNdq3b8+CBd6jmoMHD+att94qdqRlESn0Yc6WLVty0kkn8frrr/+p/MQTT+SCCy5g/fr1vPDCCyTWqsWrL77IhpUrmbt2Lfvy80muW5eTGjZEsrNJjI3lk/fe49w+fZjbogURsbG02rqVrKlTad+iBaH79pG3axeZGRnUeeUVwqOj2f/yy+R//DF7tm4lNDeXaBF03z4mXHMNbdq0ofVTT1Hnf/9DfG6M0IQE1s6ZQ1JSEvL3vyMffvinuPObNGHNd9+xdu1aetx/PzJt2sHKuDhITfV6agD9+sG8eV4yKkhsnTt7yRS804gbNhysi4iAlBS48Uav/t57YedOL1lGRh5MaP37e/UffeRdQyvo4UVEeKcvjz3Wq1+61Fs3LOzgz4Jeo6p3+tS3PsTu/6mKLKkUwZJK+cnPz2fNmjU0bdr0QNmaNWuoV68ekZGRLFq0iKZNm7Js2TLmzp3LFVdcwWeffcYTTzxBs2bNuOOOOzj++OMBuPbaa3n77bf5+uuv2bNnD02aNOHNN9/k888/p1u3bqSlpfHDDz+Q63oiDz30EGPHjmX16tXldjwNGzakW7duvP/++4D3wGn37t2ZPXs227dvL3F9ASKBKKBdy5b8lJ4OQP82bXj01ltJqFmTV156idN79uSZl17izU2bABh/ySW0jo0lKyOD2vv28ZfjjvO+1O+5h4yMDOZdeCGnt2hBVFgYOzMziY2IQFq3Zv+99/LJJ59w5rhx1Fi3jn27drFt40bqxMaSf/LJREyezB9//IG0aUPsH39AXh6S440gl3vBBcg77xAaGuolh0N6oQwdyowhQ9iSmclfzznn8IO95RZ48kkvWdWq9acqFeHbU06hxeuv0yQiAlq3PphwCpLP3Xd7yXD1arRfP/JECIuMPNjuzjvh3HNh+XL0xhshJAQJCztYf9NN3o0kS5Z41/Z8k1poKFx9NbRr5yXEt98+WF6Q9AYOhCZNvPovvjh4g0qB88/3/g0WLYLvvz+4XkgIufn5hJ1/PtSu7e1//vwDdQfanXGG1zNdtsw79ap6cALvD4WwMPjtt8LrL7jA+zl7ttfjVvWuLebne+tdcolX/9VXXn1BXX6+d0PNkCFe/XvvwYoVXvmFF0Lz5iX+HhfFkkoRLKlULGvXriU2Npa4uDj2799PeHh4kW1zc3OZOXMm3bt3JyQkhKysLNatW0d0dLT35TtvHk2aNCEiIoLatWvTo0cPsrOzWb16NT/99BPff/89kyZN4pprrmHevHmcddZZvPDCC4SFhdGgQQNycnJYsGABeXl5nH322eTm5jJ9+nQaN27MxRdfTLt27Zg6dSovvfQSzZs3p2vXrrzzzjvFHl/z5s1ZuXLlEX0m8fHx9OzZk7p16/LRRx+xySWfgl5f//79eeONNxg9ejRPPfUUjRo1Ij4+noULF/6pl3feeeexbt06Zs2aBXinMUfccgsntm7N5Zddxt8GDSInJ4eT4+O5qH9/Zn//Pc8+9hgRwKCRI7ns8cfJzc3lYuCUk0/mH0OHsn3LFpYtWcKP27fTbcQI5s6Ywb7HH6fN8cfT4thjCQV2bt/OLR99xJ6uXbn6ggtImTKFtscfz7y0NCQvjy4pKazr0YMNJ57IqKFDufLXXwkFzunbl1Bgf3Y2k+rV47XMTP49eDAJd9zBtsxMYiIjqREVRXzNmqy58UZ2n3oqrTZvJvqqq8jLySFMhFBVQlT5efhwRk2fzt/CwrhuaiHjzH77LQsTEtg1dixdn3nm8Pq0NDQlhQ2jRtGw4M5IHx89+ihbExL465IlJD766OHrr1/P8l27+KxLF4YX8gfJmoULadK2LYwYAWPGHL6++3fcO2gQ0W+99aeqveHhjH3sMX7++WfGbNhAwx9++FN9Xv365K9ZQ1ZWFvWGDEE+/dSr+Owz6HP07zEsbVJBVavV1KlTJzXV1/bt2/+0nJ+ff9jy4sWLDyv3tWfPngPzQ4cO1Q4dOuiUKVN0/fr1OnfuXL3lllv0xhtv1N27d2t+fr7ec889Cuhxxx2ngIaEhOh7772nmzdv1hUrVmjPnj21R48eOmHCBD333HML3iWkgNaoUUNTUlIOrHfxxRdrWFjYgfqoqCht3br1geXw8PAD89HR0X/a1pFOPXr00NGjRx9YbtCggYrIUW0rNDS0VO3q1q1bppiLmkJAT+nWTR9/4AF98z//0b90766ARoAeA5p4yNSja1dt3bq1RoPWB20ImgTaBLQpaJjb7jGg3WvX1nagHUCTQbuGhenpf/mL4tbrDHpqTIx2Ah3cvr12BE2Ij9fevXvrozfdpEM6d9ZuNWro3QMG6EXt2umpiYkaFxenV111lTaPjtaUGjX0tv79tQVoMxdDwXHFgzYU0dPattUEF09tn/oH775bJ44fr106dNABf/2rbt68+aj/7wBpWorvWOupGFMGBf9/pITbnXNycoiMjERV2bhxIw0aNChyezk5OeTl5XHLLbdw9dVX07lzZ/Lz89m6dSsJCQl8+eWX/Pvf/2bx4sW888479OzZk++//54333yTZ599lrfffps+ffqwf/9+Zs2axamnnkpsbCwAixcv5t///jeTJ08GoEOHDowcOZIPP/yQ7OxsBg4cyL59+1i1ahX/+te/CA0NJTc3l4cffphff/2V9u3b06FDB9q3b8+4ceNYunQpu3fvpnv37px00kmsWLGCn3/+mb///e+kpaVx7LHHsmzZMqKjozn55JPp37//gVOYANdccw1z5syh4P9kSEgIXbp0YejQoWRnZ/Puu+/Su3dvRowYwSWXXMKPP/5Iamoqu3fvZvny5Vx44YXUr1+fE044geXLlzNhwgQ2btzIyy+/zObNm/nkk0947rnneOWVV3j77bdZunRpkf9GISEhxMTEsHPnTsLDw4mOjqZbt25s2rSJHj16cPbZZ/PMM88QHx/PypUr6du3L+vWrePrr79mxYoVADz22GPcfvvtf9puXFwc8fHxrFq1CoDu3buTk5PDzp07Wb58OdHR0YSHh7Njxw4AOnfuTJ06dfj8888PbCM6OprU1FR+8OmVnHXWWYSGhrJs2TLS3anWq666iuTkZDIzM/nPf/7D5s3ea6liYmJo3bo1M2fO9E55HgU7/VUESyrGQFZWFg888AD/93//R0JCQsD2u379etasWUN8fDyNGzcmOjoaEWHhwoXUrVsXEaFOnTolJumi5OXlkZOTQ40aNQ6rU1UWLVrEhg0byMjIYMWKFdx+++3UqlWLvLw8du3adWC/tQqe2yplHCtWrCA/P59WrVrx1VdfsW7dOurUqUODBg3o2LHjgRtQtm/fTnx8/IFYf/rpJ5o1a0ZYWBhPPvkkw4YNo2XLlogIixcvJiEhgdq1a5OXl0d0dDRz585l+vTpDBo0iFruWtaOHTtYs2YN7dq1+1O82dnZPPbYY7Rs2ZK//e1vREZGHtVnWsCSShEsqRhjzJGzoe+NMcYEXKVPKiLSR0SWiki6iBTyhJsxxphAqdRJRURCgeeBvkBb4GIRaRvcqIwxpvqq1EkF6AKkq+pKVd0HTAQGBDkmY4yptip7UmkErPVZznBlfyIiw0QkTUTSMjMzAxacMcZUN5U9qZSKqo5V1VRVTU1MTAx2OMYYU2VV9qSyDmjss5zkyowxxgRBZU8qs4FWItJMRCKAgcCUIMdkjDHVVqV/+FFE+gFPAaHAeFV9sIT2mcDvR7m7BGDLUa5bWdkxVw92zNVDWY75WFUt8fpBpU8qgSQiaaV5orQqsWOuHuyYq4dAHHNlP/1ljDGmArGkYowxptxYUjkyY4MdQBDYMVcPdszVg9+P2a6pGGOMKTfWUzHGGFNuLKkYY4wpN5ZUSqEqD68vIuNFZLOILPApO0ZEponIcvcz3pWLiDzjPof5IpISvMiPjog0FpFvRGSRiCwUkZtceVU+5igRmSUiv7pjvteVNxORme7YJrkHiBGRSLec7uqbBjP+shCRUBGZKyIfu+UqfcwislpEfhOReSKS5soC+rttSaUE1WB4/deAPoeUjQS+UtVWwFduGbzPoJWbhgEvBijG8pQL3KqqbYFuwPXu37MqH3MO0EtVTwSSgT4i0g14BBijqi2BbcAQ134IsM2Vj3HtKqubgMU+y9XhmE9T1WSf51EC+7utqjYVMwHdgak+y3cBdwU7rnI+xqbAAp/lpUADN98AWOrm/wNcXFi7yjoBHwFnVJdjBmoAvwBd8Z6sDnPlB37PgalAdzcf5tpJsGM/imNNwvsS7QV8DEg1OObVQMIhZQH93baeSslKNbx+FVNPVTe4+Y1APTdfpT4Ld4qjIzCTKn7M7jTQPGAzMA1YAWxX1VzXxPe4Dhyzq/8DqBPYiMvFU8AdQL5brkPVP2YFvhCROSIyzJUF9Hc7rKwbMFWbqqqIVLn7zkWkJvAecLOq7hCRA3VV8ZhVNQ9IFpE44AOgdZBD8isR+SuwWVXniMipwY4ngHqo6joRqQtME5ElvpWB+N22nkrJquPw+ptEpAGA+7nZlVeJz0JEwvESyluq+r4rrtLHXEBVtwPf4J36iRORgj8sfY/rwDG7+tpAVoBDLauTgXNEZDXeG2F7AU9TtY8ZVV3nfm7G++OhCwH+3bakUrLqOLz+FOByN3853nWHgvLB7q6RbsAfPt3qSkG8Lsk4YLGqPulTVZWPOdH1UBCRaLxrSIvxksv5rtmhx1zwWZwPfK3upHtloap3qWqSqjbF+z/7tapeShU+ZhGJEZHYgnngTGABgf7dDvaFpcowAf2AZXjnof8v2PGU87G9A2wA9uOdUx2Cdy75K2A58CVwjGsreHfCrQB+A1KDHf9RHG8PvPPO84F5bupXxY+5AzDXHfMC4B5X3hyYBaQD/wUiXXmUW0539c2DfQxlPP5TgY+r+jG7Y/vVTQsLvqsC/bttw7QYY4wpN3b6yxhjTLmxpGKMMabcWFIxxhhTbiypGGOMKTeWVIwxxpQbSyrGlEBE6onI2yKy0g1/MV1EzgtSLKeKyEk+y/8QkcHBiMWYwtgwLcYUwz0s+SEwQVUvcWXHAuf4cZ9henB8qkOdCuwCfgZQ1Zf8FYcxR8OeUzGmGCJyOt7Dgj0LqQsFHsb7oo8EnlfV/7ixpkbjjXTbHpgDDFJVFZFOwJNATVd/hapuEJFv8R7E7IH3QOoy4G4gAm+4kEuBaGAGkAdkAjcCpwO7VPVxEUkGXsIbiXgFcJWqbnPbngmcBsQBQ1T1h/L7lIw5yE5/GVO8dnhDxRdmCN7QFp2BzsDVItLM1XUEbsZ7B09z4GQ35tizwPmq2gkYDzzos70IVU1V1SeAH4FuqtoRb+yqO1R1NV7SGKPe+zIOTQyvA3eqage8J6RH/X97d8zaVBQFcPx/aJVSkDjo5uDkUugodPAztOAsxbGIqx3tJ2ihRXB1LHZqpyIIXUVLlwwuiuLmUEU6GCjH4d6Hl2IipS/b/7fk5ea+5GQIJ+/cx7nNa7OZeb/G9BxpSix/SZcQES8oVxMj4AuwGBFdL6kBZcOjEfAuM7/Vc04oe9b8oFy5vKldkWcoLXI6u83xHWC3NgC8Dnz+T1wD4GZmHtWhV5S2I52uceaHGos0FSYVabIh8LB7kplPIuIW8B74CjzNzMP2hFr++t0MnVN+awEMM3NpzGedNcc7wGZm7jfltKvo4ulikabC8pc02VtgLiLWmrH5+ngIrNWyFhFxr3aHHecjcDsilur8axGxMGbugL9tyFeb8V/AjYuTM/MncBoRD+rQI+Do4jxp2vzHIk1QF9dXgK2IeEZZID8D1inlpbvAcb1L7DuwMuG9RrVUtl3LVbOU3QmH/5i+AbyOiFNKYuvWag6AvYhYpizUt1aBlxExD3wCHl/+G0tX491fkqTeWP6SJPXGpCJJ6o1JRZLUG5OKJKk3JhVJUm9MKpKk3phUJEm9+QPtNl/YtWu/kAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# Plot loss (MSE) over time\n",
    "plt.plot(loss_vec, 'k-', label='Train Loss')\n",
    "plt.plot(test_loss, 'r--', label='Test Loss')\n",
    "plt.title('Loss (MSE) per Generation')\n",
    "plt.legend(loc='upper right')\n",
    "plt.xlabel('Generation')\n",
    "plt.ylabel('Loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here is how to calculate the model accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "On predicting the category of low birthweight from regression output (<2500g):\n",
      "Test Accuracy: 0.5\n",
      "Train Accuracy: 0.6225165562913907\n"
     ]
    }
   ],
   "source": [
    "# Model Accuracy\n",
    "actuals = np.array([x[0] for x in birth_data])\n",
    "test_actuals = actuals[test_indices]\n",
    "train_actuals = actuals[train_indices]\n",
    "test_preds = [x[0] for x in sess.run(final_output, feed_dict={x_data: x_vals_test})]\n",
    "train_preds = [x[0] for x in sess.run(final_output, feed_dict={x_data: x_vals_train})]\n",
    "test_preds = np.array([1.0 if x < 2500.0 else 0.0 for x in test_preds])\n",
    "train_preds = np.array([1.0 if x < 2500.0 else 0.0 for x in train_preds])\n",
    "# Print out accuracies\n",
    "test_acc = np.mean([x == y for x, y in zip(test_preds, test_actuals)])\n",
    "train_acc = np.mean([x == y for x, y in zip(train_preds, train_actuals)])\n",
    "print('On predicting the category of low birthweight from regression output (<2500g):')\n",
    "print('Test Accuracy: {}'.format(test_acc))\n",
    "print('Train Accuracy: {}'.format(train_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate new points on the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Data Predictions: [1. 1.]\n"
     ]
    }
   ],
   "source": [
    "# Need new vectors of 'AGE', 'LWT', 'RACE', 'SMOKE', 'PTL', 'HT', 'UI'\n",
    "new_data = np.array([[35, 185, 1., 0., 0., 0., 1.],\n",
    "                     [18, 160, 0., 1., 0., 0., 1.]])\n",
    "new_data_scaled = np.nan_to_num(normalize_cols(new_data, train_max, train_min))\n",
    "new_logits = [x[0] for x in sess.run(final_output, feed_dict={x_data: new_data_scaled})]\n",
    "new_preds = np.array([1.0 if x < 2500.0 else 0.0 for x in new_logits])\n",
    "\n",
    "print('New Data Predictions: {}'.format(new_preds))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
